Determining Popularity Ratings Using Social and Interactive Applications for Mass Media

ABSTRACT

Systems, methods, devices, and computer program products provide social and interactive applications for mass media based on real time ambient-audio and/or video identification. In some implementations, a method includes: receiving descriptors identifying ambient audio associated with a media broadcast; comparing the descriptors to one or more reference descriptors; and determining a rating for the media broadcast based at least in part on the results of the comparison.

RELATED APPLICATIONS

This application is a continuation and claims priority under 35 USC §120to U.S. patent application Ser. No. 13/842,471, filed Mar. 15, 2013,which is a continuation of and claims priority to U.S. patentapplication Ser. No. 11/563,653, filed on Nov. 27, 2006, which claimsthe benefit of priority from U.S. Provisional Patent Application No.60/740,760, filed Nov. 29, 2005, and U.S. Provisional Patent ApplicationNo. 60/823,881, filed Aug. 29, 2006, the entire contents of which arehereby incorporated by reference.

This application is related to U.S. patent application Ser. No.11/563,661, entitled “Social and Interactive Applications For MassMedia,” filed Nov. 27, 2006, and U.S. patent application Ser. No.11/563,665, entitled “Detecting Repeating Content In Media,” filed Nov.27, 2006 (now U.S. Pat. No. 7,991,770 issued Aug. 2, 2011). Each ofthese patent applications is incorporated by reference herein in itsentirety.

TECHNICAL FIELD

The disclosed implementations are related to social and interactiveapplications for mass media.

BACKGROUND

Mass media channels (e.g., television and radio broadcasts) typicallyprovide limited content to a large audience. By contrast, the World WideWeb provides vast amounts of information that may only interest a fewindividuals. Conventional interactive television attempts to bridgethese two communication mediums by providing a means for viewers tointeract with their televisions and to receive content and/or servicesrelated to television broadcasts.

Conventional interactive television is typically only available toviewers through cable or satellite networks for a subscription fee. Toreceive interactive television service the viewer has to rent or buy aset-top box and have it installed by a technician. The viewer'stelevision is connected to the set-top box, which enables the viewer tointeract with the television using a remote control or other inputdevice, and to receive information, entertainment and services (e.g.,advertisements, online shopping, forms and surveys, games andactivities, etc.).

While conventional interactive television can improve the viewer'stelevision experience, there remains a need for social and interactiveapplications for mass media that do not rely on significant additionalhardware or physical connections between the television or radio and aset-top box or computer.

One social and interactive television application that is lacking withconventional and interactive television systems is the ability toprovide complementary information to the mass media channel in aneffortless manner. With conventional systems, a user would have tolog-on to a computer and query for such information which would diminishthe passive experience offered by mass media. Moreover, conventionaltelevision systems cannot provide complementary information in real-timewhile the user is watching a broadcast.

Another deficiency in conventional television systems and interactivetelevision systems is a simple method to assess the popularity ofbroadcasting events. The popularity ratings of broadcasting events areof high interest to users, broadcasters and advertisers. These needs arepartially resolved by measurement systems like the Nielsen® ratings.These ratings, however, require dedicated hardware installation andcooperation from participating viewers.

SUMMARY

The deficiencies described above are addressed by the disclosed systems,methods, apparatuses, user interfaces and computer program products forproviding social and interactive applications based on real-timeambient-audio and/or video identification.

In some implementations, a method includes: receiving descriptorsidentifying ambient audio associated with a media broadcast; comparingthe descriptors to one or more reference descriptors; and determining arating for the media broadcast based at least in part on the results ofthe comparison.

In some implementations, a method includes: generating descriptorsidentifying ambient audio associated with a media broadcast; providingthe descriptors to a ratings provider for determining a rating for themedia broadcast based on the descriptors; receiving the rating from theratings provider; and displaying the rating on a display device.

In some implementations, a method includes: recording ambient audiosnippets from a media broadcast; generating a descriptor from theambient audio snippets; and providing the descriptor to a ratingsprovider.

In some implementations, a system includes a database of referencedescriptors. A server is operatively coupled to the database and to aclient system. The server is configurable to receive a descriptor fromthe client system for identifying ambient audio associated with a mediabroadcast, comparing the received descriptor with one or more referencedescriptors, and determining a rating for the media broadcast based atleast in part on the results of the comparison.

In some implementations, a system includes an audio detectorconfigurable for sampling ambient audio. A client interface isoperatively coupled to the audio detector and configurable to generatedescriptors identifying a media broadcast. The client interface isconfigurable for transmitting the descriptors to a network resource, andfor receiving rating information from the network resource based on thedescriptors.

Other implementations are directed to systems, methods, apparatuses,user interfaces, and computer program products.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of one embodiment of a mass personalizationsystem.

FIG. 2 illustrates one embodiment of an ambient-audio identificationsystem, including the client-side interface shown in FIG. 1.

FIG. 3 is a flow diagram of one embodiment of a process for providingmass-personalization applications.

FIG. 4 is a flow diagram of one embodiment of an audio fingerprintingprocess.

FIG. 5 is a flow diagram of one embodiment of a user interface forinteracting with mass personalization applications.

FIG. 6 is a block diagram of one embodiment of hardware architecture fora client system for implementing the client-side interface shown in FIG.1.

FIG. 7 is a flow diagram of one embodiment of a repetition detectionprocess.

DETAILED DESCRIPTION Mass Personalization Applications

Mass personalization applications provide personalized and interactiveinformation related to mass media broadcasts (e.g., television, radio,movies, Internet broadcasts, etc.). Such applications include but arenot limited to: personalized information layers, ad hoc social peercommunities, real-time popularity ratings and video (or audio)bookmarks, etc. Although some of the mass media examples disclosedherein are in the context of television broadcasts, the disclosedimplementations are equally applicable to radio and/or music broadcasts.

Personalized information layers provide complementary information to themass media channel. Examples of personalized information layers includebut are not limited to: fashion, politics, business, health, traveling,etc. For example, while watching a news segment on a celebrity, afashion layer is presented to the viewer on a television screen or acomputer display device, which provides information and/or imagesrelated to the clothes and accessories the celebrity is wearing in thenews segment. Additionally, personalized layers may includeadvertisements promoting products or services related to the newssegment, such as a link to a clothing store that is selling clothes thatthe celebrity is wearing.

Ad hoc social peer communities provide a venue for commentary betweenusers who are watching the same show on television or listening to thesame radio station. For example, a user who is watching the latest CNNheadlines can be provided with a commenting medium (e.g., a chat room,message board, wiki page, video link, etc.) that allows the user tochat, comment on or read other viewers responses to the ongoing massmedia broadcast.

Real-time popularity ratings provide content providers and users withratings information (similar to Nielsen ratings). For example, a usercan instantaneously be provided with real-time popularity ratings oftelevision channels or radio stations being watched or listened to bythe user's social network and/or by people with similar demographics.

Video or audio bookmarks provide users with low effort ways of creatingpersonalized libraries of their favorite broadcast content. For example,a user can simply press a button on a computer or a remote controldevice and a snippet of ambient audio and/or video of the broadcastcontent is recorded, processed and saved. The snippet can be used as abookmark to refer to the program, or portions of the program, for laterviewing. The bookmark can be shared with friends or saved for futurepersonal reference.

Mass Personalization Network

FIG. 1 is a block diagram of a mass personalization system 100 forproviding mass personalization applications. The system 100 includes oneor more client-side interfaces 102, an audio database server 104 and asocial application server 106, all of which communicate over a network108 (e.g., the Internet, an intranet, LAN, wireless network, etc.).

A client interface 102 can be any device that allows a user to enter andreceive information, and which is capable of presenting a user interfaceon a display device, including but not limited to: a desktop or portablecomputer; an electronic device; a telephone; a mobile phone; a displaysystem; a television; a computer monitor; a navigation system; aportable media player/recorder; a personal digital assistant (PDA); agame console; a handheld electronic device; and an embedded electronicdevice or appliance. The client interface 102 is described more fullywith respect to FIG. 2.

In some implementations, the client-interface 102 includes an ambientaudio detector (e.g., a microphone) for monitoring and recording theambient audio of a mass media broadcast in a broadcast environment(e.g., a user's living room). One or more ambient audio segments or“snippets” are converted into distinctive and robust statisticalsummaries, referred to as “audio fingerprints” or “descriptors.” In someimplementations, the descriptors are compressed files containing one ormore audio signature components that can be compared with a database ofpreviously generated reference descriptors or statistics associated withthe mass media broadcast.

A technique for generating audio fingerprints for music identificationis described in Ke, Y., Hoiem, D., Sukthankar, R. (2005), ComputerVision for Music Identification, In Proc. Computer Vision and PatternRecognition, which is incorporated herein by reference in its entirety.In some implementations, the music identification approach proposed by(hereinafter “Ke et al.”) is adapted to generate descriptors fortelevision audio data and queries, as described with respect to FIG. 4.

A technique for generating audio descriptors using wavelets is describedin U.S. Provisional Patent Application No. 60/823,881, for “AudioIdentification Based on Signatures.” That application describes atechnique that uses a combination of computer-vision techniques andlarge-scale-data-stream processing algorithms to create compactdescriptors/fingerprints of audio snippets that can be efficientlymatched. The technique uses wavelets, which is a known mathematical toolfor hierarchically decomposing functions.

In “Audio Identification Based on Signatures,” an implementation of aretrieval process includes the following steps: 1) given the audiospectra of an audio snippet, extract spectral images of, for example,11.6*w ms duration, with random spacing averaging d-ms apart. For eachspectral image: 2) compute wavelets on the spectral image; 3) extractthe top-t wavelets; 4) create a binary representation of the top-twavelets; 5) use min-hash to create a sub-fingerprint of the top-twavelets; 6) use LSH with b bins and I hash tables to findsub-fingerprint segments that are close matches; 7) discardsub-fingerprints with less than v matches; 8) compute a Hamming distancefrom the remaining candidate sub-fingerprints to the querysub-fingerprint; and 9) use dynamic programming to combined the matchesacross time.

In some implementations, the descriptors and an associated useridentifier (“user id”) for identifying the client-side interface 102 aresent to the audio database server 104 via network 108. The audiodatabase server 104 compares the descriptor to a plurality of referencedescriptors, which were previously determined and stored in an audiodatabase 110 coupled to the audio database server 104. In someimplementations, the audio database server 104 continuously updates thereference descriptors stored in the audio database 110 from recent massmedia broadcasts.

The audio database server 104 determines the best matches between thereceived descriptors and the reference descriptors and sends best-matchinformation to the social application server 106. The matching processis described more fully with respect to FIG. 4.

In some implementations, the social application server 106 acceptsweb-browser connections associated with the client-side interface 102.Using the best-match information, the social application server 106aggregates personalized information for the user and sends thepersonalized information to the client-side interface 102. Thepersonalized information can include but is not limited to:advertisements, personalized information layers, popularity ratings, andinformation associated with a commenting medium (e.g., ad hoc socialpeer communities, forums, discussion groups, video conferences, etc.).

In some implementations, the personalized information can be used tocreate a chat room for viewers without knowing the show that the viewersare watching in real time. The chat rooms can be created by directlycomparing descriptors in the data streams transmitted by client systemsto determine matches. That is, chat rooms can be created around viewershaving matching descriptors. In such an implementation, there is no needto compare the descriptors received from viewers against referencedescriptors.

In some implementations, the social application server 106 serves a webpage to the client-side interface 102, which is received and displayedby a web browser (e.g., Microsoft Internet Explorer™) running at theclient-side interface 102. The social application server 106 alsoreceives the user id from the client-side interface 102 and/or audiodatabase server 104 to assist in aggregating personalized content andserving web pages to the client-side interface 102.

It should be apparent that other implementations of the system 100 arepossible. For example, the system 100 can include multiple audiodatabases 110, audio database servers 104 and/or social applicationservers 106. Alternatively, the audio database server 104 and the socialapplication server 106 can be a single server or system, or part of anetwork resource and/or service. Also, the network 108 can includemultiple networks and links operatively coupled together in varioustopologies and arrangements using a variety of network devices (e.g.,hubs, routers, etc.) and mediums (e.g., copper, optical fiber, radiofrequencies, etc.). Client-server architectures are described hereinonly as an example. Other computer architectures are possible.

Ambient Audio Identification System

FIG. 2 illustrates an ambient audio identification system 200, includinga client-side interface 102 as shown in FIG. 1. The system 200 includesa mass media system 202 (e.g., a television set, radio, computer,electronic device, mobile phone, game console, network appliance, etc.),an ambient audio detector 204, a client-side interface 102 (e.g., adesktop or laptop computer, etc.) and a network access device 206. Insome implementations, the client-side interface 102 includes a displaydevice 210 for presenting a user interface (UI) 208 for enabling a userto interact with a mass personalization application, as described withrespect to FIG. 5.

In operation, the mass media system 202 generates ambient audio of amass media broadcast (e.g., television audio), which is detected by theambient audio detector 204. The ambient audio detector 204 can be anydevice that can detect ambient audio, including a freestandingmicrophone and a microphone that is integrated with the client-sideinterface 102. The detected ambient audio is encoded by the client-sideinterface 102 to provide descriptors identifying the ambient audio. Thedescriptors are transmitted to the audio database server 104 by way ofthe network access device 206 and the network 108.

In some implementations, client software running at the client-sideinterface 102 continually monitors and records n-second (e.g., 5 second)audio files (“snippets”) of ambient audio. The snippets are thenconverted into m-frames (e.g., 415 frames) of k-bit encoded descriptors(e.g., 32-bit), according to a process described with respect to FIG. 4.In some implementations, the monitoring and recording is event based.For example, the monitoring and recording can be automatically initiatedon a specified date and at a specified time (e.g., Monday, 8:00 P.M.)and for a specified time duration (e.g., between 8:00-9:00 P.M.).Alternatively, the monitoring and recording can be initiated in responseto user input (e.g., a mouse click, function key or key combination)from a control device (e.g., a remote control, etc.). In someimplementations, the ambient audio is encoded using a streamingvariation of the 32-bit/frame discriminative features described in Ke etal.

In some implementations, the client software runs as a “side bar” orother user interface element. That way, when the client-side interface102 is booted up, the ambient audio sampling can start immediately andrun in the “background” with results (optionally) being displayed in theside bar without invoking a full web-browser session.

In some implementations, the ambient audio sampling can begin when theclient-side interface 102 is booted or when the viewer logs into aservice or application (e.g., email, etc.)

The descriptors are sent to the audio database server 104. In someimplementations, the descriptors are compressed statistical summaries ofthe ambient audio, a described in Ke et al. By sending statisticalsummaries, the user's acoustic privacy is maintained because thestatistical summaries are not reversible, i.e., the original audiocannot be recovered from the descriptor. Thus, any conversations by theuser or other individuals monitored and recorded in the broadcastenvironment cannot be reproduced from the descriptor. In someimplementations, the descriptors can be encrypted for extra privacy andsecurity using one or more known encryption techniques (e.g., asymmetricor symmetric key encryption, elliptic encryption, etc.).

In some implementations, the descriptors are sent to the audio databaseserver 104 as a query submission (also referred to as a querydescriptor) in response to a trigger event detected by the monitoringprocess at the client-side interface 102. For example, a trigger eventcould be the opening theme of a television program (e.g., opening tuneof “Seinfeld”) or dialogue spoken by the actors. In someimplementations, the query descriptors can be sent to the audio databaseserver 104 as part of a continuous streaming process. In someimplementations, the query descriptors can be transmitted to the audiodatabase server 104 in response to user input (e.g., via remote control,mouse clicks, etc.).

Mass Personalization Process

FIG. 3 is a flow diagram a mass personalization process 300. The stepsof process 300 do not have to be completed in any particular order andat least some steps can be performed at the same time in amulti-threading or parallel processing environment.

The process 300 begins when a client-side interface (e.g., client-sideinterface 102) monitors and records snippets of ambient audio of a massmedia broadcast in a broadcast environment (302). The recorded ambientaudio snippets are encoded into descriptors (e.g., compressedstatistical summaries), which can be sent to an audio database server(304) as queries. The audio database server compares the queries againsta database of reference descriptors computed from mass media broadcaststatistics to determine candidate descriptors that best match the query(308). The candidate descriptors are sent to a social application serveror other network resource, which uses the candidate descriptors toaggregate personalized information for the user (310). For example, ifthe user is watching the television show “Seinfeld,” then querydescriptors generated from the show's ambient audio will be matched withreference descriptors derived from previous “Seinfeld” broadcasts. Thus,the best matching candidate descriptors are used to aggregatepersonalized information relating to “Seinfeld” (e.g., news stories,discussion groups, links to ad hoc social peer communities or chatrooms, advertisements, etc.). In some implementations, the matchingprocedure is efficiently performed using hashing techniques (e.g.,direct hashing or locality sensitive hashing (LSH)) to achieve a shortlist of candidate descriptors, as described with respect to FIG. 4. Thecandidate descriptors are then processed in a validation procedure, suchas described in Ke et al.

In some implementations, query descriptors from different viewers aredirectly matched rather than matching each query with a database ofreference descriptors. Such an embodiment would enable the creation ofad hoc social peer communities on subject matter for which a database ofreference descriptors is not available. Such an embodiment could matchin real-time viewers who are in the same public form (e.g., stadium,bar, etc.) using portable electronic devices (e.g., mobile phones, PDAs,etc.).

Popularity Ratings

In some implementations, real-time and aggregate statistics are inferredfrom a list of viewers currently watching the broadcast (e.g., show,advertisement, etc.). These statistics can be gathered in the backgroundwhile viewers are using other applications. Statistics can include butare not limited to: 1) the average number of viewers watching thebroadcast; 2) the average number of times viewers watched the broadcast;3) other shows the viewers watched; 4) the minimum and peak number ofviewers; 5) what viewers most often switched to when they left abroadcast; 6) how long viewers watch a broadcast; 7) how many timesviewers flip a channel; 8) which advertisements were watched by viewers;and 9) what viewers most often switched from when they entered abroadcast, etc. From these statistics, one or more popularity ratingscan be determined.

The statistics used to generate popularity ratings can be generatedusing a counter for each broadcast channel being monitored. In someimplementations, the counters can be intersected with demographic groupdata or geographic group data. The popularity ratings can be used byviewers to “see what's hot” while the broadcast is ongoing (e.g., bynoticing an increased rating during the 2004 Super Bowl half-timeperformance). Advertisers and content providers can also use popularityratings to dynamically adjust the material shown in response to ratings.This is especially true for advertisements, since the short unit lengthand numerous versions of advertisements generated by advertisingcampaigns are easily exchanged to adjust to viewer rating levels. Otherexamples of statistics include but are not limited to: popularity of atelevision broadcast versus a radio broadcast by demographics or time,the popularity of times of day, i.e., peak watching/listening times, thenumber of households in a given area, the amount of channel surfingduring particular shows (genre of shows, particular times of day), thevolume of the broadcast, etc.

The personalized information is sent to the client-side interface (312).The popularity ratings can also be stored in a database for use by otherprocesses (318), such as the dynamic adjustment of advertisementsdescribed above. The personalized information is received at theclient-side interface (314) where it is formatted and presented in auser interface (316). The personalized information can be associatedwith a commenting medium (e.g., text messages in a chat room) that ispresented to the user in a user interface. In some implementations, achat room can include one or more subgroups. For example, a discussiongroup for “Seinfeld” might include a subgroup called “Seinfeld Experts,”or a subgroup may be associated with a particular demographic, such aswomen between the ages of 20-30 who watch “Seinfeld,” etc.

In some implementations, the raw information (e.g., counter values) usedto generate statistics for popularity ratings is collected and stored atthe client-side interface rather than at the social application server.The raw information can be transferred to the broadcaster whenever theuser is online and/or invokes a mass personalization application.

In some implementations, a broadcast measurement box (BMB) is installedat the client-side interface. The BMB can be a simple hardware devicethat is similar to a set-top box but does not connect to the broadcastdevice. Unlike the Neilsen rating system, which requires hardware to beinstalled in the television, the BMB can be installed near the massmedia system or within the range of the television signal. In someimplementations, the BMB automatically records audio snippets andgenerates descriptors, which are stored in memory (e.g., flash media).In some implementations, the BMB can optionally include one or morehardware buttons which can be pressed by a user to indicate whichbroadcast they are watching (similar to Neilsen® ratings). The BMBdevice can be picked-up by the ratings provider from time to time tocollect the stored descriptors, or the BMB can broadcast the storeddescriptors to one or more interested parties over a network connection(e.g., telephone, Internet, wireless radio, such as short messageservice (SMS) for radio, etc.) from time to time.

In some implementations, advertisements can be monitored to determinethe ad's effectiveness, which can be reported back to advertisers. Forexample, which ads were watched, skipped, volume level of the ads, etc.

In some implementations, an image capture device (e.g., digital camera,video recorder, etc.) can be used to measure how many viewers arewatching or listening to a broadcast. For example, various knownpattern-matching algorithms can be applied to an image or a sequence ofimages to determine the number of viewers present in a broadcastenvironment during a particular broadcast. The images and or dataderived from the images can be used in combination with audiodescriptors to gather personalized information for a user, computepopularity ratings, or for any other purpose.

Audio Fingerprinting Process

FIG. 4 is a flow diagram of audio fingerprinting process 400. The stepsof process 400 do not have to be completed in any particular order andat least some steps can be performed at the same time in amulti-threading or parallel processing environment. The process 400matches query descriptors generated at a client-side interface (e.g.,client-side interface 102) to reference descriptors stored in one ormore databases in real-time and with low latency. The process 400 adaptsa technique proposed by Ke et al. to handle ambient audio data (e.g.,from a television broadcast) and queries.

The process 400 begins at a client-side interface by decomposing ambientaudio snippets (e.g., 5-6 seconds of audio) of a mass media broadcastcaptured by an ambient audio detector (e.g., microphone) intooverlapping frames (402). In some implementations, the frames are spacedapart by several milliseconds (e.g., 12 ms apart). Each frame isconverted into a descriptor (e.g., a 32-bit descriptor) that is trainedto overcome audio noise and distortion (404), as described in Ke et al.In some implementations, each descriptor represents an identifyingstatistical summary of the audio snippet.

In some implementations, the descriptors can be sent as query snippets(also referred to as query descriptors) to an audio database serverwhere they are matched to a database of reference descriptorsidentifying statistical summaries of previously recorded audio snippetsof the mass media broadcast (406). A list of candidate descriptorshaving best matches can be determined (408). The candidate descriptorscan be scored, such that candidate descriptors that are temporallyconsistent with the query descriptor are scored higher than candidatedescriptors that are less temporally consistent with the querydescriptor (410). The candidate descriptors with the highest scores(e.g., score exceeds a sufficiently high threshold value) aretransmitted or otherwise provided to a social application server (412)where they can be used to aggregate personalized information related tothe media broadcast. Using a threshold ensures that descriptors aresufficiently matched before the descriptors are transmitted or otherwiseprovided to the social application server (412).

In some implementations, the database of reference descriptors can begenerated from broadcasts given by various media companies, which can beindexed and used to generate the descriptors. In other implementations,reference descriptors can also be generated using television guides orother metadata and/or information embedded in the broadcast signal.

In some implementations, speech recognition technology can be used tohelp identify which program is being watched. Such technology could helpusers discuss news events instead of just television shows. For example,a user could be watching a Shuttle launch on a different channel thananother viewer and, therefore, possibly getting a different audio signal(e.g., due to a different newscaster). Speech recognition technologycould be used to recognize keywords (e.g., Shuttle, launch, etc.), whichcan be used to link the user with a commenting medium.

Hashing Descriptors

Ke et al. uses computer vision techniques to find highly discriminative,compact statistics for audio. Their procedure trained on labeled pairsof positive examples (where x and x′ are noisy versions of the sameaudio) and negative examples (where x and x′ are from different audio).During this training phase, machine-learning technique based on boostinguses the labeled pairs to select a combination of 32 filters andthresholds that jointly create a highly discriminative statistic. Thefilters localize changes in the spectrogram magnitude, using first andsecond order differences across time and frequency. One benefit of usingthese simple difference filters is that they can be calculatedefficiently using a integral image technique described in Viola, P. andJones, M. (2002), Robust Real-Time Object Detection, InternationalJournal of Computer Vision, which is incorporated by reference herein inits entirety.

In some implementations, the outputs of these 32 filters are thresholds,giving a single bit per filter at each audio frame. These 32 thresholdresults form only transmitted descriptors of that frame of audio. Thissparsity in encoding ensures the privacy of the user to unauthorizedeavesdropping. Further, these 32-bit descriptors are robust to the audiodistortions in the training data, so that positive examples (e.g.,matching frames) have small Hamming distances (i.e., distance measuringdiffering number of bits) and negative examples (e.g., mismatchedframes) have large Hamming distances. It should be noted that more orfewer filters can be used and more than one bit per filter can be usedat each audio frame (e.g., more bits using multiple threshold tests).

In some implementations, the 32-bit descriptor itself used as a hash keyfor direct hashing. The descriptor is a well-balanced hash function.Retrieval rates are further improved by querying not only the querydescriptor, but also a small set of similar descriptors (up to a Hammingdistance of 2 from the original query descriptor).

Within-Query Temporal Consistency

Once the query descriptors are matched to the audio database using thehashing procedure described above, the matches are validated todetermine which of the database return hits are accurate matches.Otherwise, a candidate descriptor might have many frames matched to thequery descriptor but with the wrong temporal structure.

In some implementations, validation is achieved by viewing each databasehit as support for a match at a specific query-database offset. Forexample, if the eight descriptor (q_(g)) in a 5-second, 415-frame-long“Seinfeld” query snippet, q, hits the 1008^(th) database descriptor(x₁₀₀₈), this supports a candidate match between the 5-second query andframes 1001 through 1415 in the audio database. Other matches betweenq_(n) and x_(1000+n) (1<n<415) would support this same candidate match.

In addition to temporal consistency, we need to account for frames whenconversations temporarily drown out the ambient audio. This can bemodeled as an exclusive switch between ambient audio and interferingsounds. For each query frame i, there is a hidden variable, y_(i): ify_(i)=0, the i^(th) frame of the query is modeled as interference only;if y_(i)=1, the i^(th) frame is modeled as from clean ambient audio.Taking an extreme view (pure ambient or pure interference) is justifiedby the extremely low precision with which each audio frame isrepresented (32 bits) and softened by providing additional bit-flopprobabilities for each of the 32 positions of the frame vector undereach of the two hypotheses (y_(i)=0 and y_(i)=1). Finally, we model thebetween-frame transitions between ambient-only and interference-onlystates as a hidden first-order Markov process, with transitionprobabilities derived from training data. For example, we can re-use the66-parameter probability model given by Ke et al., CVPR 2005.

The final model of the match probability between a query vector, q, andan ambient-database vector at an offset of N frames, x_(N), is:

P(q|x ^(N))=Π_(n=1) ⁴¹⁵ P(<q _(n) ,x _(N+n) >|y _(n))P(y _(n) |y_(n−1))  (1)

where <q_(n), x_(m)> denotes the bit differences between the 32-bitframe vectors q_(n) and x_(m). This model incorporates both the temporalconsistency constraint and the ambient/interference hidden Markov model.

Post-Match Consistency Filtering

People often talk with others while watching television, resulting insporadic but strong acoustic interference, especially when usinglaptop-based microphones for sampling the ambient audio. Given that mostconversational utterances are two or three seconds in duration, a simplecommunication exchange between viewers could render a 5-second queryunrecognizable.

In some implementations, post-match filtering is used to handle theseintermittent low-confidence mismatches. For example, we can use acontinuous-time hidden Markov model of channel switching with anexpected dwell time (i.e., time between channel changes) of L seconds.The social application server 106 indicates the highest-confidence matchwithin the recent past (along with its “discounted” confidence) as partof state information associated with each client session. Using thisinformation, the server 106 selects either the content-index match fromthe recent past or the current index match, base on whichever has thehigher confidence.

We use M_(h) and C_(h) to refer to the best match for the previous timestep (5 seconds ago) and its log-likelihood confidence score. If wesimply apply the Markov model to this previous best match, withouttaking another observation, then our expectation is that the best matchfor the current time is that same program sequence, just 5 secondsfurther along, and our confidence in this expectation is C_(h)-l/L,where 1=5 seconds is the query time step. This discount of l/L in thelog-likelihood corresponds to the Markov model probability, e^(−l/L), ofnot switching channels during the l-length time step.

An alternative hypothesis is generated by the audio match for thecurrent query. We use Mo to refer to the best match for the currentaudio snippet: that is, the match that is generated by the audiofingerprinting process 400. Co is the log-likelihood confidence scoregiven by the audio fingerprinting process 400.

If these two matches (the updated historical expectation and the currentsnippet observation) give different matches, we select the hypothesiswith the higher confidence score:

$\begin{matrix}{\left\{ {M_{0},C_{0}} \right\} = \left\{ {\begin{matrix}\left\{ {M_{h},{C_{h} - {l/L}}} \right\} & {{{{if}\mspace{14mu} C_{h}} - {l/L}} > C_{0}} \\\left. {M_{0},C_{0}} \right\} & {otherwise}\end{matrix},} \right.} & (2)\end{matrix}$

where M₀ is the match that is used by the social application server 106for selecting related content and M₀ and C₀ are carried forward on thenext time step as M_(h) and C_(h).

User Interface

FIG. 5 is a flow diagram of one embodiment of a user interface 208 forinteracting with mass personalization applications. The user interface208 includes a personalized layer display area 502, a commenting mediumdisplay area 504, a sponsored links display area 506 and a contentdisplay area 508. The personalized layer display area 502 providescomplementary information and/or images related to the video contentshown in the content display area 508. The personalized layers can benavigated using a navigation bar 510 and an input device (e.g., a mouseor remote control). Each layer has an associated label in the navigationbar 510. For example, if the user selects the “Fashion” label, then thefashion layer, which includes fashion related content associated with“Seinfeld,” will be presented in the display area 502.

In some implementations, the client-side interface 102 includes adisplay device 210 capable of presenting the user interface 208. In someimplementations, the user interface 208 is an interactive web pageserved by the social application server 106 and presented in a browserwindow on the screen of the display device 210. In some implementations,the user interface 208 is persistent and will be available forinteraction after the broadcast audio used in the content match processhas shifted in time. In some implementations, the user interface 208 isdynamically updated over time or in response to a trigger event (e.g., anew person enters the chat room, a commercial begins, etc.). Forexample, each time a commercial is broadcast, the sponsored linksdisplay area 506 can be updated with fresh links 518 related to thesubject matter of the commercial.

In some implementations, the personalized information and sponsoredlinks can be emailed to the viewer or shown on a side bar at a latertime.

In some implementations, the client-side interface 102 receivespersonalized information from the social application server 106. Thisinformation can include a web page, email, a message board, links,instant message, a chat room, or an invitation to join an ongoingdiscussion group, eRoom, video conference or netmeeting, voice call(e.g., Skype®), etc. In some implementations, the user interface 208provides access to comments and/or links to comments from previouslyseen broadcasts or movies. For example, if user is currently watching aDVD of “Shrek” he may want to see what people said about the movie inthe past.

In some implementations, the display area 502 includes a rating region512, which is used to display popularity ratings related to a broadcast.For example, the display area 512 may show how many viewers arecurrently watching “Seinfeld” compared to another television show thatis broadcast at the same time.

In some implementations, the commenting medium display area 504 presentsa chat room type environment where multiple users can comment aboutbroadcasts. In some implementations, the display area 504 includes atext box 514 for inputting comments that are sent to the chat room usingthe input mechanism 516 (e.g., a button).

The sponsored links display area 506 includes information, images and/orlinks related to advertising that is associated with the broadcast. Forexample, one of the links 518 may take the user to a web site that isselling “Seinfeld” merchandise.

The content display area 508 is where the broadcast content isdisplayed. For example, a scene from the current broadcast can bedisplayed with other relevant information (e.g., episode number, title,timestamp, etc.). In some implementations, the display area 508 includescontrols 520 (e.g., scroll buttons) for navigating through the displayedcontent.

Video Bookmarks

In some implementations, a button 522 is included in the content displayarea that can be used to bookmark video. For example, by clicking thebutton 522, the “Seinfeld” episode shown in the display area 508 isadded to the user's favorites video library, which can then be viewedon-demand through a web-based streaming application or other accessmethods. According to the policy set by the content owner, thisstreaming service can provide free single-viewing playback, collectpayments as the agent for the content owners, or insert advertisementsthat would provide payment to the content owners.

Client-Side Interface Hardware Architecture

FIG. 6 is block diagram of hardware architecture 600 for the client-sideinterface 102 shown in FIG. 1. Although the hardware architecture 600 istypical of a computing device (e.g., a personal computer), the disclosedimplementations can be realized in any device capable of presenting auser interface on a display device, including but not limited to:desktop or portable computers; electronic devices; telephones; mobilephones; display systems; televisions; monitors; navigation systems;portable media players/recorders; personal digital assistants; gamesystems; handheld electronic devices; and embedded electronic devices orappliances.

In some implementations, the system 600 includes one or more processors602 (e.g., CPU), optionally one or more display devices 604 (e.g., CRT,LCD, etc.), a microphone interface 606, one or more network interfaces608 (e.g., USB, Ethernet, FireWire® ports, etc.), optionally one or moreinput devices 610 (e.g., mouse, keyboard, etc.) and one or morecomputer-readable mediums 612. Each of these components is operativelycoupled to one or more buses 614 (e.g., EISA, PCI, USB, FireWire@,NuBus, PDS, etc.).

In some implementations, there are no display devices or input devicesand the system 600 just performs sampling and encoding (e.g., generatingdescriptors, etc.) in the background without user input.

The term “computer-readable medium” refers to any medium thatparticipates in providing instructions to a processor 602 for execution,including without limitation, non-volatile media (e.g., optical ormagnetic disks), volatile media (e.g., memory) and transmission media.Transmission media includes, without limitation, coaxial cables, copperwire and fiber optics. Transmission media can also take the form ofacoustic, light or radio frequency waves.

The computer-readable medium(s) 612 further includes an operating system616 (e.g., Mac OS®, Windows®, Unix, Linux, etc.), a networkcommunications module 618, client software 620 and one or moreapplications 622. The operating system 616 can be multi-user,multiprocessing, multitasking, multithreading, real-time and the like.The operating system 616 performs basic tasks, including but not limitedto: recognizing input from input devices 610; sending output to displaydevices 604; keeping track of files and directories on storage devices612; controlling peripheral devices (e.g., disk drives, printers, imagecapture device, etc.); and managing traffic on the one or more buses614.

The network communications module 618 includes various components forestablishing and maintaining network connections (e.g., software forimplementing communication protocols, such as TCP/IP, HTTP, Ethernet,USB, FireWire®, etc.).

The client software 620 provides various software components forimplementing the client-side of the mass personalization applicationsand for performing the various client-side functions described withrespect to FIGS. 1-5 (e.g., ambient audio identification). In someimplementations, some or all of the processes performed by the clientsoftware 620 can be integrated into the operating system 616. In someimplementations, the processes can be at least partially implemented indigital electronic circuitry, or in computer hardware, firmware,software, or in any combination thereof.

Other applications 624 can include any other software application,including but not limited to: word processors, browsers, email, InstantMessaging, media players, telephony software, etc.

Detecting Advertisements and Rebroadcasts Repetition Detection

When preparing a database for search, it helps to be able to pre-flagrepeated material using the descriptors previously described. Repeatingmaterial can include but is not limited to repeating shows,advertisements, sub-segments (e.g., stock footage in news shows), etc.Using these flags, repeated material can be presented in a way that doesnot push all other material beyond the attention span of a userconducting a search (e.g., beyond the first 10-20 hits). The process 700described below provides a way to detect those duplicates prior to anysearch queries on the database.

Video Ad Removal

One of the complaints that broadcasters have had about allowing materialto be searched and played back is the rebroadcast of embeddedadvertising. From the point of view of the broadcasters, thisrebroadcast is counterproductive: it lowers the value of the broadcaststhat the advertiser pays for directly, since it provides that advertiserwith free advertising. Unless old advertisements are removed and newadvertisements are put in place in a way that returns some review to theoriginal broadcasters, they do not profit from the replay of theirpreviously broadcast material. The process 700 described below providesa way of detecting embedded advertisement by looking for repetitions,possibly in conjunction with other criteria (e.g., duration, volume,visual activity, bracketing blank frames, etc.).

Video Summarization

If a “summary” (i.e., shorter version) of non-repeated program materialis needed, one way to get that is to remove the advertisements (asdetected by repeated material) and to take segments from the materialjust preceding and just following the advertisement location. Onbroadcast television, these positions in the program typically contain“teasers” (before the ads) and “recaps” (just after the ads). If asummary is to be made of a news program that includes a mix ofnon-repeated and repeated non-advertisement material, typically therepeated non-advertisement material corresponds to a sound bite. Thesesegments generally contribute less information than the anchorperson'snarration of the news story and are good candidates for removal. If asummary is to be made of a narrative program (e.g. a movie or a serialinstallment), repeated audio tracks typically correspond to themesounds, mood music, or silence. Again, these are typically good segmentsto remove from a summary video. The process 700 described below providesa way of detecting these repeated audio tracks so they can be removedfrom the summary video.

Repetition Detection Process

FIG. 7 is a flow diagram of one embodiment of a repetition detectionprocess 700 in accordance. The steps of process 700 do not have to becompleted in any particular order and at least some steps can beperformed at the same time in a multi-threading or parallel processingenvironment.

The process 700 begins by creating a database of audio statistics from aset of content such as television feeds, video uploads, etc. (702). Forexample, the database could contain 32-bit/frame descriptors, asdescribed in Ke et al. Queries are taken from the database and runagainst the database to see where repetitions occur (704). In someimplementations, a short segment of audio statistics is taken as a queryand run checked for non-identity matches (matches that are notidentical) using hashing techniques (e.g. direct hashing or localitysensitive hashing (LSH)) to achieve a short list of possible auditorymatches. These candidate matches are then processed in a validationprocedure, for example, as described in Ke, et al. Content correspondingto a validated candidate match can be identified as repeating content(706).

The non-identity matches that are strongest are “grown” forwards andbackwards in time, to find the beginning and ending points of therepeated material (708). In some implementations, this can be done usingknown dynamic programming techniques (e.g., Viterbi decoding). Inextending the match forward in time, the last time slice in the strong“seed” match is set as “matching” and the last time slice of the firstbelow-believable-strength match for the same database offset between thequery and the match is set as “not matching.” In some implementations,match scores for individual frames in between these two fixed points areused as observations, and a first-order Markov model allowing withinstate transitions, plus a single transition from “matching” to“not-matching” states, is used. The transition probability from matchingto not matching to 1/L can be set somewhat arbitrarily, where L is thenumber of frames between these two fixed points, corresponding to theleast knowledge of the transition location within the allowed range.Another possibility for selecting transition probabilities would use thematch strength profiles to bias this estimate to an earlier or latertransition. But this would increase the complexity of the dynamicprogramming model and is not likely to improve the results, since thematch strengths are already used as observations within this period. Thesame process is used to grow the segment matches backwards in time(e.g., just switch past/future and run the same algorithm).

In some implementations the audio cues are combined with non-auditoryinformation (e.g., visual cues) to obtain higher matching accuracies.For example, the matches that are found with audio matching can then beverified (or checked a second time) by using simple visual similaritymetrics (710). These metrics can include but are not limited to: colorhistograms (e.g., frequencies of similar colors in two images),statistics on number and distribution of edges, etc. These need not becomputed only over the entire image, but can be computed for sub-regionsof the images as well, and compared to the corresponding sub-regions inthe target image.

For those applications that are looking for advertisements (in contrastwith all types of repeated material), the results of repeated-materialdetection can be combined with metrics aimed at distinguishingadvertisements from non-advertisements (712). These distinguishingcharacteristics can rely on advertising conventions, such as durations(e.g., 10/15/30-second spots are common), on volume (e.g.,advertisements tend to be louder than surrounding program material, soif the repeated material is louder than the material on either side, itis more likely to be an advertisement), on visual activity (e.g.,advertisements tend to have more rapid transitions between shots andmore within-shot motion, so if the repeated material has larger framedifferences than the material on either side, it is more likely to be anadvertisement), and on bracketing blank frames (locally insertedadvertisements typically do not completely fill the slot that is leftfor it by the national feed, resulting in black frames and silence at aspacing that is a multiple of 30 seconds).

Once advertisements are identified, material surrounding theadvertisements can be analyzed and statistics can be generated. Forexample, statistics can be generated about how many times a particularproduct is advertised using a particular creative (e.g., images, text),or how many times a particular segment is aired, etc. In someimplementations, one or more old advertisements can be removed orreplaced with new advertisements. Additional techniques foradvertisement detection and replacement are described in Covell, M.,Baluja, S., Fink, M., Advertisement Detection and Replacement UsingAcoustic and Visual Repetition, IEEE Signal Processing Society, MMSP2006 International Workshop on Multimedia Signal Processing, Oct. 3-6,2006, BC Canada, which article is incorporated by reference herein inits entirety.

In some implementations, information from content owners about thedetailed structure of the content (e.g., where ad material was inserted,where programs were repeated) could be used to augment the process 700and increase matching accuracies. In some implementations, videostatistics can be used to determine repetition instead of audio. Inother implementations, a combination of video and audio statistics canbe used.

Audio Snippet Auctions

In some implementations, advertisers can participate in auctions relatedto the presence of ambient audio that is related to the product orservice that the advertiser want to sell. For example, multipleadvertisers could bid in an auction for the right to associate itsproducts or services with an audio snippet or descriptor associated with“Seinfeld.” The winner of the auction could then put some relatedinformation in front of the viewer (e.g., the sponsored links) wheneverthe subject ambient audio is present. In some implementations,advertisers could bid on ambient audio snippets having a meta-leveldescription. For example, advertisers could bid on audio that isassociated with a television ad (e.g., this is the audio associated witha Ford Explorer TV ad), on closed captioning (e.g., the captioning says“Yankees baseball”), on program segment location (e.g., this audio willoccur 15 min into the “Seinfeld” and will occur 3 minutes after theprevious commercial break and 1 min before the next commercial break),or on low-level acoustic or visual properties (e.g., “background music,”“conversational voices,” “explosive-like”, etc.)

In some implementations, one or more mass personalization applicationscan be run in the background while the user performs other tasks such asbrowsing another web site (e.g., a sponsored link). Material that isrelated to a media broadcast (e.g., television content) can participatein the same sponsored link auctions as material that is related toanother content source (e.g., web site content). For example, TV relatedads can be mixed with ads that correspond to the content of a currentweb page.

Various modifications may be made to the disclosed implementations andstill be within the scope of the following claims.

1. A method, comprising: receiving, by a computing device, user input tocapture one or more ambient audio snippets of media played proximate tothe computing device; causing the computing device to capture the one ormore ambient audio snippets of the media, in response to the user input;encoding, by the computing device, the one or more ambient audiosnippets that are captured; sending, by the computing device, the one ormore encoded ambient audio snippets to a server; receiving, by theserver, the one or more encoded ambient audio snippets from thecomputing device; matching, by the server, at least one of the one ormore encoded ambient audio snippets with at least one of a plurality ofpre-stored audio fingerprints to obtain information descriptive of themedia; sending, by the server, the information descriptive of the media,to the computing device; and causing the information descriptive of themedia to be displayed on a user interface of the computing device. 2.The method of claim 1, further comprising determining, by the server,popularity of the media based, in part, on the matching of the at leastone of the one or more encoded ambient audio snippets.
 3. The method ofclaim 1, wherein the media include at least one of a piece of music, atelevision show, and a movie, and wherein the at least one of thepre-stored audio fingerprints correspond to the at least one of thepiece of music, the television show, and the movie.
 4. The method ofclaim 1, further comprising, determining an identification of the media,based at least in part on the matched at least one of the one or moreencoded audio snippets, and wherein the information descriptive of themedia includes an identification of the media.
 5. The method of claim 1,wherein the one or more encoded ambient audio snippets are insufficientto recover the media.
 6. A method, comprising: receiving, by a computingdevice, user input to capture one or more ambient audio snippets ofmedia played in an environment of a client device; capturing, by thecomputing device, the one or more ambient audio snippets of the media,in response, at least in part, to the user input; encoding, by thecomputing device, the one or more ambient audio snippets that arecaptured; sending, by the computing device, the one or more encodedambient audio snippets to a server, for matching at least one of the oneor more encoded ambient audio snippets with at least one of a pluralityof pre-stored audio fingerprints to obtain information descriptive ofthe media; receiving, by the computing device, the informationdescriptive of the media from the server; and causing the informationdescriptive of the media to be displayed on a user interface of thecomputing device.
 7. The method of claim 6, wherein the user interfaceenables a user of the computing device to share the informationdescriptive of the media, with one or more other users.
 8. The method ofclaim 6, wherein the one or more encoded ambient audio snippets areinsufficient to recover the media.
 9. The method of claim 6, wherein themedia is at least one of a television show, movie, music, and video, andthe information descriptive of the media includes one or more of a titleand an episode number of the media.
 10. The method of claim 6, whereinthe user input is initiated by user activation of a control element ofthe user interface.
 11. The method of claim 6, wherein the capturing isfurther in response to detecting a triggering event, by the computingdevice.
 12. The method of claim 11, wherein the triggering eventincludes at least one of music and talk indicative of the media.
 13. Themethod of claim 6, wherein the capturing is scheduled for at least oneof a particular date, time, and duration.
 14. The method of claim 6,wherein the capturing is performed by using a microphone of thecomputing device.
 15. A system, comprising: one or more processors, andone or more non-transitory computer-readable media having instructionsstored thereon that, when executed by the one or more processors, causeperformance of operations comprising: receiving user input to captureone or more ambient audio snippets of media played in an environment ofthe system; capturing the one or more ambient audio snippets of themedia, in response to the user input; encoding the one or more ambientaudio snippets that are captured; sending the one or more encodedambient audio snippets to a server, for matching at least one of the oneor more encoded ambient audio snippets with at least one of a pluralityof pre-stored audio fingerprints to obtain information descriptive ofthe media; receiving the information descriptive of the media from theserver; and causing the information descriptive of the media to bedisplayed on a user interface of the system.
 16. The system of claim 15,wherein the user interface enables a user of the system to share theinformation descriptive of the media, with one or more other users. 17.The system of claim 15, wherein the one or more encoded ambient audiosnippets are insufficient to recover the media.
 18. The system of claim15, wherein the media is at least one of a television show, movie,music, and video, and the information descriptive of the media includesone or more of a title and an episode number of the media.
 19. Thesystem of claim 15, wherein the user input is initiated by useractivation of a control element of the user interface.
 20. The system ofclaim 15, wherein the capturing is further in response to detecting atriggering event, by the system.
 21. The system of claim 20, wherein thetriggering event includes at least one of music and talk indicative ofthe media.
 22. The system of claim 15, wherein the capturing isscheduled for at least one of a particular date, time and duration. 23.The system of claim 15, wherein the capturing is performed using amicrophone of the system.
 24. A method, comprising: receiving, by acomputing device, a user input to capture one or more ambient audiosnippets of a media played in an environment associated with a clientdevice; capturing, by the computing device, the one or more ambientaudio snippets of the media, in response to the user input; encoding, bythe computing device, the one or more ambient audio snippets that arecaptured; sending, by the computing device, the one or more encodedambient audio snippets to a server for matching at least one of the oneor more encoded ambient audio snippets, with at least one of a pluralityof pre-stored audio fingerprints to obtain a matched audio snippet;receiving, by the computing device, a request to share a representationof the media with one or more users, wherein the representation is basedon the matched audio snippet; and transmitting, by the computing device,instructions to cause sharing of the representation of the media withthe one or more users.
 25. The method of claim 24, wherein therepresentation is a portion of the media that is of a pre-definedlength.
 26. The method of claim 24, wherein the representation issimilar to the one or more ambient audio snippets of the media.
 27. Themethod of claim 24, wherein sharing of the representation of the mediaincludes sharing a link to the representation of the media with the oneor more users.
 28. The method of claim 24, wherein the instructions tocause sharing of the representation of the media further includeinstructions to cause sharing of a user comment related to the media.29. The method of claim 24, wherein a user of the computing device andthe one or more users are part of a same social network.
 30. The methodof claim 29, wherein the transmitting of the instructions to causesharing causes at least the part of the representation of the media tobe provided on a platform associated with the same social network.